Parallel Algorithms for Predictive Modelling

Parallel computing enables the analysis of very large data sets using large collections of flexible models with many variables. The computational methods are based on ideas from computational linear algebra and can draw on the extensive research on parallel algorithms in this area. Many algorithms for the direct and iterative solution of penalised least squares problems and for updating can be applied. Both methods for dense and sparse problems are applicable. An important property of the algorithms is their scalability, i.e., their ability to solve larger problems in the same time using hardware which grows linearly with the problem size. While in most cases large granularity parallelism is to be preferred, it turns out that even smaller granularity parallelism can be exploited effectively in the problems considered. The development is illustrated by four examples of nonparametric regression techniques. In a first example, additive models are considered. While the backfitting method contains dependencies which inhibit parallel execution it turns out that parallelisation over the data leads to a viable method, akin to the bagging algorithm without replacement which is known to have superior statistical properties in many cases. The second example considers radial basis function fitting with thin plate splines. Here the direct approach turns out to be non-scalable but an approximation with finite elements is shown to be scalable and parallelises well. One of the most popular algorithms in data mining is MARS (Multivariate Adaptive Regression Splines). This is discussed in the third example. MARS has been modified to use a multiscale approach and a parallel algorithm with a small granularity has been seen to give good results. The final example considers the current research area of sparse grids. Sparse grids take up many ideas from the previous examples and, in fact, can be considered as a generalisation of MARS and additive models. They are naturally parallel when the combination technique is used. We discuss limitations and improvements of the combination technique.